Multiplex scheme conversion apparatus

ABSTRACT

A multiplex scheme conversion apparatus ( 103 ) includes (i) a packet loss judgment unit 202 that judges the packet loss of a TS packet, performs error processing in the case where a packet loss is detected, and outputs the payload data on which error processing has already been performed and (ii) a Box making unit 206 that makes ‘moov’ including error information and ‘mdat’ based on inputted AU data and the display time in a PES header.

BACKGROUND OF THE INVENTION

(1) Field of the Invention

The present invention relates to a multiplex scheme conversion apparatusfor converting the multiplex scheme of the coded media data such asvideo and sound and outputting the converted coded media data.

(2) Description of the Related Art

With a recent increase in the capacity of storage media andcommunication networks or the development of communication technique,apparatuses and services for processing coded multimedia data such asvideo and sound have become popular. For example, in the broadcastingfield, conventional analog broadcasting is replaced by digitally codedmedia data broadcasting. The digital broadcasting is received by onlyimmobile apparatuses at present, but in the future, broadcasting formobile apparatuses such as mobile phones is scheduled to be started.Also, in the communication field, the environment where multi media datais handled by both immobile terminals and mobile terminals is beingprepared. For example, a video distribution service for third generationmobile phones has already been started. In light of the presentsituation, the use method of recording content data received viabroadcasting or the Internet on memory cards such as secure digital (SD)cards or optical discs such as digital versatile disk-rewritable(DVD-RAM) and causing such apparatuses to share the content data isexpected to become popular.

At the time of distributing media data via broadcasting or networks, orby using a storage medium, the header information and the media datanecessary for reproducing media data are multiplexed. In multiplexing, amultiplex scheme is standardized respectively for broadcasting, storageapparatuses such as DVDs, and mobile apparatuses. First, for digitalbroadcasting or DVDs, the Moving picture expert group (MPEG-2) systemstandard that is standardized by the International standardizationorganization/international engineering consortium (ISO/IECJTC1/SC29/WG11) is used. Also, for mobile terminals, the MP4 file formatstandardized by the ISO/IEC JTC1/SC29/WG11 is employed in theTransparent end-to-end packet switched streaming service (TS26.234)prescribed, as a wireless video distribution standard, by the Thirdgeneration partnership project (3GPP) that is the internationalstandardization group that is an establishment for standardizing thethird generation mobile communication system.

Also, the MPEG-4 advanced video coding (AVC) is standardized as thestandard that succeeds the MPEG-2 Visual or the MPEG-4 visual that arepopular as a video coding method at present. Therefore, it is expectedthat coded video data of the MPEG-4 AVC is multiplexed, broadcast,stored or distributed using the MPEG-2 system standard or the MP4 fileformat (called MP4 from here).

The outline of coded data multiplex scheme in the MPEG-2 system and theMP4 will be described below. In the MPEG-2 system and the MPEG-4, thebasic unit used in handling coded data is an access unit (AU), and thestructure of an AU will be described first. An AU as a unit includescoded data for one picture, and the AU data in the MPEG-4 AVC has thestructure shown in FIG. 1A to 1C. In the MPEG-4 AVC, it is possible toinclude, in AU data, not only the data necessary for decoding picturesbut also supplementary information called supplemental enhancementinformation (SEI) that is unnecessary for decoding, AU boundaryinformation and the like. All the data is stored in the networkadaptation layer (NAL) unit. Note that, in the MPEG-2 system, a NAL unitcalled access unit delimiter indicating the start of an AU is surelyadded to the top of the AU. The NAL unit is composed of a header and apayload as shown as FIG. 1A. The header size is 1 byte and the headerincludes a field indicating the type of data stored in the payload(called NAL unit type from here). NAL unit type values are defined basedon the respective kinds of data such as a slice and a SEI, a NAL unittype is referred to when obtaining a data type stored in the NAL unit. ANAL unit such as a header information and a SEI are stored in an AU inaddition to slice data for one picture as shown as FIGS. 1B and 1C. Asinformation for identifying the boundary of NAL unit data is notincluded in the NAL unit, boundary information can be added to the topof each NAL unit when an AU is stored. Boundary information is addedusing the following two types of addition methods: adding a start codeprefix shown in 3 bytes of 0x000001 (called byte stream format fromhere) in FIG. 1B; and adding the size of NAL unit as shown in FIG. 1C(called NAL size format from here). Note that it is prescribed that atleast one zero_byte whose value is 0x00that is represented in 1 byte isadded before the start code prefix of the leading NAL unit and a NALunit having a specific NAL unit type value of an AU. The byte streamformat is used in the MPEG-2 system, and the NAL size format is used inthe MP4.

Next, a slice and header information will be described in detail. Slicesare roughly divided into two types: Instantaneous decoder refresh (IDR)slices; and the other type of slices. An IDR slice is slice data that isintra-coded, and header information such as later-described sequenceparameter set (SPS) and the like can be switched only in such an IDRslice. In the case where an IDR slice is included in a picture, theother slices in the picture are IDR slices, therefore, an AU includingIDR slices is called as an IDR AU from here. Also, a unit composed ofAUs from an IDR AU to the AU immediately before the next IDR AU iscalled sequence. Here, random accesses are performed, on asequence-by-sequence basis because only AUs in a sequence are referredto in decoding slice data of an AU. Next, there are two types of headerinformation: SPS; and picture parameter set (PPS). SPS is headerinformation that is fixed on a sequence-by-sequence basis and PPS isheader information that is switchable on a picture-by-picture basis.Header information can include several SPSs and PPSs, and these SPSs andPPSs are distinguished from each other based on index numbers. Also, oneSPS or PPS is stored in one NAL unit. Index numbers of an SPS and a PPSthat are referred to by each picture are obtained in the following way.First, the index number of a PPS referred to by a picture is shown inthe header part of slice data. Next, as the index number of an SPSreferred to by the PPS is shown in the PPS, the index number of the SPSreferred to by the picture can be obtained by analyzing the PPS referredto by the picture. An SPS and PPS referred to by a picture is necessaryin decoding the slice data of a picture.

Next, boundary information addition method in broadcasting at the timeof separating AU data by the MPEG-2 system will be described. In theMPEG-2 system, the coded data is multiplexed into a packetizedelementary stream (PES) packet and then the PES packet is multiplexedinto a transport stream (TS) packet. FIG. 2A shows the structure of aPES packet, and FIG. 2B shows the structure of a PES packet and a TSpacket. In the payload of a PES packet, access unit (AU) data is stored.FIG. 2A (1) to (3) show a storage example indicating how AU data isstored in the payload of a PES packet. Several AUs may be storedaltogether as shown in FIG. 2A(1) and 2A(2) and AU data may be dividedand stored as shown in FIG. 2A(3). Further, in the payload, staffingdata can be included separately from the AU data. The header of the PESpacket starts from a 4-byte start code composed of a start code prefixshown in 3 bytes of 0x000001 and a stream ID represented in 1 byte. Thestream ID is an identification number indicating the type of a codeddata included in the payload data of the PES packet, and theidentification number may be an arbitrary value between 0xE0 and 0XEFinclusive in the MPEG-4 AVC. It is possible to store, in the header,decoding time and display time of the leading AU in the payload, butsuch pieces of time information are not always stored in all PESpackets, in other words, there are PES packets where no time informationis stored. In the case where pieces of time information being thedecoding time or the display time, on AU are not shown by the header ofa PES packet but needed, AU data is analyzed and the differential valuein decoding time or display time of the current AU and the immediatelybefore AU is obtained. Note that the starting position in a PES packetis detected by searching 4-byte start code in the payload data of TSpackets. On the other hand, the data of the PES packet is divided andstored in the payloads of TS packets as shown in FIG. 2B. A TS packet isa packet having a fixed length of 188 bytes, and it is composed of a4-byte header, an adaptation field and payload data. Note that anadaptation field is included only in the case where a specific flag isset in the header. In the header, an identification number called PIDindicating the type of data transmitted by the TS packet and a countercalled continuity_counter is included. The continuity_counter is a 4-bitfield. In the case of TS packets having an identical PID, values areincremented one by one according to the sending order of such TS packetsuntil values are incremented to be the maximum value and then re-countedfrom the starting value. The association between the PID of a TS packetand the type of data transmitted by the TS packet is separately providedin program information transmitted by the TS packet. Therefore, at thetime of receiving TS packets, PIDs of TS packets are obtained first, andthen these packets are divided depending on PID values. For example, inthe case where the program information obtained at the starting time ofreceiving shows that the MPEG-4 AVC data is transmitted by the TS packetwhose PID is 32, obtaining the TS packet whose PID is 32 makes itpossible to obtain the AU data of the MPEG-4 AVC. Here, a gap betweencontinuity_counter values of the received TS packets indicates that apacket loss has occurred in a transmission path. In addition, in thecase where AU data is separated from a TS packet, a PES packet isseparated from the payload data of a TS packet and the AU data isseparated from the separated PES packet.

Lastly, the multiplex scheme of AU data in the MP4 will be described. Inthe MP4, header information on a sample-by-sample basis or media data ismanaged on an object-by-object basis. The object is called Box. Here, asample is a basic unit in handling media data in the MP4, and one samplecorresponds to 1 AU. A sample number is assigned to each sample in anascending order of decoding time, and these sample numbers areincremented by one for each sample. FIG. 3A shows the structure of theBox composed of the following fields: (i) size that is the size of thewhole Box including a size field; (ii) type that is a Box identifier,the identifier basically being four alphabets, the field length being 4bytes, and the Box in the MP4 file being searched by judging whether thesequential 4-byte data matches the identifier of the type field; (iii)version that is the version number of the Box; (iv) flags that is theflag information set for each Box; and (v) data that is data such asheader information or media data.

Note that some Boxes do not include fields of version and flags becausethey are unnecessary. Identifiers of type fields are used in referringto Boxes in the following description, for example, the Box whose typeis ‘moov’ is called ‘moov’. The Box structure in the MP4 file is shownas FIG. 3B. The MP4 file is composed of ‘fytp’, ‘moov’ and, ‘mdat’ or‘moof’, and ‘fytp’ is placed in the top of the MP4 file. Information foridentifying an MP4 file is included in ‘fytp’, and media data is storedin ‘mdat’. Each media data included in ‘mdat’ is called ‘trak’, and each‘trak’ is identified by a ‘trak’ ID. Next, header information on asample included in each ‘trak’ of ‘mdat’ is stored in ‘moov’. In ‘moov’,as shown as FIG. 4A, Boxes are hierarchically placed, and pieces ofheader information of audio media tracks and video media tracks areseparately stored in a ‘trak’. In a ‘trak’, Boxes are hierarchicallyplaced, and the following information is stored in each Box in ‘stbl’:(i) sizes, decoding time and display starting time of samples; or (ii)information on randomly-accessible samples (FIG. 4B). Suchrandomly-accessible samples are called Sync samples, and a list ofsample numbers of the Sync samples is shown by ‘stss’ in ‘stbl’. Theheader information of all the samples in a ‘trak’ is stored in ‘moov’ inthe above description, but it is possible to divide these ‘trak’s intofragments and store the header information on a fragment-by-fragmentbasis. The header information on each unit obtained by dividing the‘trak’ is shown in ‘moof’. In the example of FIG. 5, the headerinformation of samples to be stored in ‘mdat’ #1 can be stored in ‘moof’#1.

Here is a case where broadcasting data received in a mobile terminalsuch as a mobile phone is transmitted by e-mail in a form of anattachment. In 3GPP, in the case where video and sound are handled ine-mail, it is prescribed that such media data is multiplexed using theMP4. Therefore, the TS multiplex scheme needs to be converted to the MP4at the time of transmitting e-mail. The following is the description ofhow a conventional multiplex scheme conversion apparatus converts, intothe MP4 file, the packet sequence of TS packets where coded data of theMPEG-4 AVC is multiplexed (For example, refer to Japanese Laid-OpenPatent application No. 2003-114845). Note that fragments are not used inthe converted MP4. FIG. 6 is a flow chart showing the conversionoperation. In the step 101, the PES packet data is separated, from thepayload of the inputted TS packet, by detecting the starting position ofthe PES packet. Next, in the step 102, the starting position of an AU isdetected from the payload of the separated PES packet. Step 101 and step102 are repeated until the starting position of an AU is detected (LoopA), and the AU data for 1 AU is separated at the time of the detection.The display time information shown in the PES header is obtained in thestep 103, and ‘moov’ and ‘mdat’ are made in the step 104. Processingfrom step 101 to step 104 is repeated until the processing of the lastTS packet is completed (Loop B), in other words, Loop B is completed atthe time of processing all the AUs in media data. After that, in thestep 105, data of ‘moov’ and ‘mdat’ are connected to each other, and theMP4 file is completed. FIG. 7 is a flow chart showing the processing ofstep 104 in detail. In the step 201, the header information necessaryfor making Boxes in ‘moov’ is obtained by analyzing AU data. In the step202, storage format of NAL units composing an AU is converted from abyte stream format to a NAL size format. In the step 203, Box data in‘moov’ is made based on the header information of AUs obtained in thestep 201. In the step 204, ‘mdat’ data is made. Note that a Box forstoring the header information, which includes the size and decodingtime of a sample, for each sample is completed at the time whenprocessing of all the AUs in a ‘trak’ is completed.

Depending on weather or other receiving conditions, a packet loss occursat the time of receiving TS packets to be broadcast. Especially, in acase where they are received by a mobile terminal such as a mobilephone, packet losses frequently occur. In the case of TS packets, it ispossible to detect that AU data is lost because of a packet loss bychecking whether continuity_counter value is discontinuous or not, butin the case of MP4, it is impossible to show that sample data is lost.This is the cause of the first problem that sample data cannot becorrectly decoded because data is lost at the time of reproducing theMP4 file, resulting in video data with low picture quality beingdisplayed.

Further, at the time of converting a multiplex scheme from TS to MP4,the storage format of the NAL unit must be converted from a byte streamformat to a NAL size format. In the byte stream format, the boundary ofthe NAL units is detected by detecting a start code added to the top ofthe NAL unit. However, in the case where the boundary part is lostbecause of a packet loss, the boundary cannot be detected. This makes itimpossible to separate the data of the NAL unit correctly.Conventionally, there is no prescribed storing method of the NAL unitdata, as a sample of MP4, which can be used in the case where NAL unitdata cannot be separated correctly. This is the cause of the secondproblem that data to be stored in a converted NAL unit cannot bedetermined.

SUMMARY OF THE INVENTION

Therefore, the present invention is conceived considering theabove-mentioned circumstances. An object of this invention is to providea multiplex scheme conversion apparatus that can avoid displaying thevideo data that cannot be correctly decoded to a user.

In order to achieve the above-mentioned object, the multiplex schemeconversion apparatus in the present invention (i) packet-multiplexes thecoded media data using the second scheme which is different from thefirst scheme by which the coded media data has been packet-multiplexedand (ii) outputs the coded media data that is packet-multiplexed usingthe second scheme, and includes: a packet loss judgment unit that judgeswhether or not a packet loss has occurred in the coded media data thatis packet-multiplexed using the first scheme; a loss information makingunit that makes loss information concerning a loss of data, based on aposition at which the data is lost in the coded media data, in the casewhere the packet loss judgment unit judges that a packet loss hasoccurred; and a multiplex unit that stores the loss information made bythe loss information making unit in a packet that is multiplexed usingthe second scheme and to packet-multiplex the coded media data using thesecond scheme.

For example, in a second-scheme file such as the MP4, this makes itpossible to obtain information on the samples whose data is partly lostbecause of a packet loss of a TS packet that is multiplexed using thefirst scheme that is the MPEG-2 system TS without decoding the sampledata, and therefore, it becomes possible to determine the operation ofdecoding or displaying samples in the system layer higher than, forexample, the audio or video decode unit.

Also, in a first aspect of the present invention, the multiplex schemeconversion apparatus may further include a packet separation unit thatseparates a header from a payload of a packet of the coded media datathat is packet-multiplexed using the first scheme.

Also, in a second aspect of the present invention, the multiplex schemeconversion apparatus may further include a frame separation unit thatdetects, in the payload separated by the packet separation unit, aboundary of frames in the coded media data and separate the frames, andin the multiplex scheme conversion apparatus, the loss informationmaking unit makes loss information concerning a loss of data on aframe-by-frame basis based on a position at which the data is lost inthe coded media data in the case where the packet loss judgment unitjudges that a packet loss has occurred, and the multiplex unit storesloss information made, on a frame-by-frame basis, by the lossinformation making unit, in the packet that is multiplexed using thesecond scheme. Here, the loss information making unit may make theinformation, as the loss information, indicating the area includingframes (i) at which data necessary for decoding are not lost and (ii)that are placed continuously in a decoding order.

For example, in a second-scheme file such as the MP4, this makes itpossible to specify the area where samples can be correctly decoded.This can prevent samples whose data is lost from being decoded resultingin allowing a turbulent picture to be outputted, and thus it is possibleto prevent the deterioration of reproduction quality.

Also, in a fourth aspect of the present invention, the multiplex schemeconversion apparatus may further include a table making unit that makesa table for registering a randomly-accessible frame, and in themultiplex scheme conversion apparatus, the multiplex unit judges whetheror not data necessary for decoding a frame is lost in the frame, basedon a position at which data is lost in the coded media data, at the timeof registering, in the table, a randomly-accessible frame in the codedmedia data, and, in the case where data necessary for decoding a frameis lost in the frame, the multiplex unit does not register, in thetable, the frame as a randomly-accessible frame.

As this guarantees that Sync samples can be correctly decoded, forexample, in the MP4 file that is multiplexed using the second scheme, itbecomes possible to prevent reproduction quality from deteriorating atthe time of random access.

Also, in the fourth aspect of the present invention, the multiplexscheme conversion apparatus may further include a conversion unit thatconverts a storage format of the coded media data. Here, “storageformat” has a narrower scope than “multiplex scheme” does. For example,it is a storage format of frame data.

Also, in a ninth aspect of the present invention, in the multiplexscheme conversion apparatus, the frame is composed of a sub-frame unit,and the conversion unit (i) determines sub-frame data to be stored inthe packet that is multiplexed using the second scheme and a size of thesub-frame data, based on the position at which data is lost in the codedmedia data, and (ii) converts the storage format by adding sizeinformation of the sub-frame unit to the top of each sub-frame unit.

This makes it possible to convert the AU data of the MPEG-4 AVC into aNAL size format also in the case where data is lost because of a packetloss, for example, in the MP4 file that is multiplexed using the secondscheme.

Also, the frame may be composed of a sub-frame unit, and the conversionunit may insert such sub-frames indicating the presence of a packet lossinto the position at which a packet loss has occurred in the coded mediadata and then packet-multiplex the coded media data using the secondmultiplex scheme.

In this way, for example, in a second-scheme file such as the MP4 file,detecting a NAL unit for notifying packet loss information that isinserted in the sample data enables the decode unit to identify theposition at which AU data is lost, and thus it becomes possible toperform appropriate error concealment processing.

Also, in the first aspect of the present invention, in the multiplexscheme conversion apparatus, the loss information making unit may makeinformation indicating whether or not data necessary for decoding apredetermined unit is lost, as the loss information.

In this way, for example, in a second-scheme file such as the MP4 file,it is possible to identify, in the system layer, the sample where thedata such as a Sequence parameter set (SPS) and a Picture parameter set(PPS) that is necessary for decoding is lost. Therefore, it becomespossible to prevent the deterioration of reproduction quality caused bydecoding samples whose data is partly lost resulting in outputting aturbulent display picture.

Also, the demultiplexer in the present invention decodespacket-multiplexed data of coded media data and includes: a lossinformation extraction unit that extracts, from the multiplexed data,loss information concerning a loss of data; a loss information analysisunit that determines data to be decoded in the coded media data, basedon the loss information extracted by the loss information extractionunit; and a decode unit that decodes the data to be decoded, determinedby the loss information analysis unit.

In this way, in multiplex data such as an MP4, it is possible todetermine the operations for decoding and displaying samples withreference to the Box for indicating sample loss information. Therefore,it becomes possible to prevent the deterioration of reproduction qualitycaused by decoding samples whose data is partly lost resulting inoutputting a turbulent display picture. In addition, it becomes possibleto obtain the same output result also in the case where another decodeand display unit which performs a different error concealment method isused.

Also, in a thirteenth aspect of the present invention, the demultiplexermay further include a loss information notification unit that makes anotification concerning reproduction quality to a user based on the lossinformation extracted by the loss information extraction unit.

In this way, in multiplex data such as an MP4, it is possible to notifya user of reproduction quality such as the loss rate of sample data(lost data/all data).

Further, the present invention can be realized not only as (i) amultiplex scheme conversion apparatus and a demultiplexer like this, butalso as (ii) a multiplex scheme conversion method and demultiplex schemeincluding steps that respectively correspond to the characteristic unitsof such a multiplex scheme conversion apparatus and a demultiplexer andas (iii) a program causing a computer to execute these steps. Also, sucha program can be distributed via recording media such as CD-ROMs orcommunication media such as the Internet.

As clearly described up to this point, the multiplex scheme conversionapparatus in the present invention can indicate the video data thatcannot be correctly decoded, and thus it becomes possible to prevent thedeterioration of reproduction quality.

FURTHER INFORMATION ABOUT TECHNICAL BACKGROUND TO THIS APPLICATION

The disclosure of Japanese Patent Application No. 2004-007817 filed onJan. 15, 2004 including specification, drawings and claims isincorporated herein by reference in its entirety.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other objects, advantages and features of the invention willbecome apparent from the following description thereof taken inconjunction with the accompanying drawings that illustrate a specificembodiment of the invention. In the Drawings:

FIG. 1A to 1C are a diagram showing the data structure of an AU in theMPEG-4 AVC;

FIGS. 2A and 2B are a diagram showing the data structure of a PES packetand a TS packet;

FIGS. 3A and 3B are a diagram showing the Box structure of an MP4;

FIGS. 4A and 4B are a diagram showing the hierarchical structure of‘moov’ in the MP4;

FIG. 5 is a diagram showing the use method of ‘moof’ in the MP4;

FIG. 6 is a flow chart showing the operation of a conventional multiplexscheme conversion apparatus;

FIG. 7 is a flow chart showing the operation for making an MP4 file inthe conventional multiplex scheme conversion apparatus;

FIG. 8 is a block diagram showing the whole structure of a multiplexscheme conversion apparatus in a first embodiment of the presentinvention;

FIG. 9 is a block diagram showing the structure of a conversion unit 103in the multiplex scheme conversion apparatus in the first embodiment ofthe present invention;

FIG. 10 is a flow chart showing the operation of the multiplex schemeconversion apparatus in the first embodiment of the present invention;

FIG. 11 is a block diagram showing the structure of a Box datadetermination unit in the multiplex scheme conversion apparatus in thefirst embodiment of the present invention;

FIG. 12 is a flow chart showing the operation of a Box making unit 205in the multiplex scheme conversion apparatus in the first embodiment ofthe present invention;

FIG. 13 is a flow chart showing the operation for making the header partof an MP4 by the Box making unit 205 in the multiplex scheme conversionapparatus in the first embodiment of the present invention;

FIG. 14 is a flow chart showing the operation of making an Error samplebox in the multiplex scheme conversion apparatus in the first embodimentof the present invention;

FIG. 15A to 15D are an example procedure of making the Error sample boxin the multiplex scheme conversion apparatus in the first embodiment ofthe present invention;

FIG. 16 is a flow chart showing the method of determining sample data inthe multiplex scheme conversion apparatus in the first embodiment of thepresent invention;

FIG. 17A to 17C are an example of converting sample data in the casewhere a packet loss occurs without crossing a NALU boundary in themultiplex scheme conversion apparatus in the first embodiment of thepresent invention;

FIGS. 18A and 18B are an example of converting sample data in the casewhere a packet loss occurs crossing NALU boundary in the multiplexscheme conversion apparatus in the first embodiment of the presentinvention;

FIG. 19A to 19C are an example of converting a sample data in the casewhere a packet loss occurs crossing a boundary of AUs in the multiplexscheme conversion apparatus in the first embodiment of the presentinvention;

FIG. 20 is a flow chart showing the method of making a Sync sample boxin the multiplex scheme conversion apparatus in a second embodiment ofthe present invention;

FIGS. 21A and 21B are an example procedure of making a Sync sample inthe multiplex scheme conversion apparatus in the second embodiment ofthe present invention;

FIG. 22 is a flow chart showing the operation of making an Error freeend sample box in the multiplex scheme conversion apparatus in a thirdembodiment of the present invention;

FIG. 23A to 23D are an example procedure of making the Error free endsample box in the multiplex scheme conversion apparatus in the thirdembodiment of the present invention;

FIG. 24A to 24D are an example procedure of making the Error free endsample box in the multiplex scheme conversion apparatus in the thirdembodiment of the present invention;

FIG. 25 is a block diagram showing the structure of a demultiplexer in afourth embodiment of the present invention;

FIG. 26 is a flow chart showing the operation performed at the time ofreproducing an MP4 file in the demultiplexer in the fourth embodiment ofthe present invention;

FIG. 27 is a flow chart showing the operation of the demultiplexer inthe fourth embodiment of the present invention;

FIG. 28 is a flow chart showing the operation of the demultiplexer in afifth embodiment of the present invention;

FIG. 29 is an illustration showing a use example of the demultiplexer ina sixth embodiment of the present invention; and

FIG. 30A to 30C are an illustration concerning a storage medium forstoring a program realizing, in a computer system, a multiplex schemeconversion method in the multiplex scheme conversion apparatus in eachof the above-mentioned embodiments and a demultiplex scheme in ademultiplexer.

DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

Embodiments of the present invention will be described below withreference to figures.

First Embodiment

First, the multiplex scheme conversion apparatus concerning a firstembodiment of the present invention will be described with reference toFIG. 8 to FIG. 19. Here, TS packets where coded video data of the MPEG-4AVC are multiplexed are inputted, after that, AU data are separated fromthe TS packets, multiplexed into an MP4 and outputted. Note that thecoded data multiplexed into packets is not limited to the MPEG-4 AVC,video coded data such as the MPEG-4 visual and H. 263, the VC-1 that ison the stage of standardization by the Society of motion picture andtelevision engineers (SMPTE), or audio coded data such as the MPEG-4 AACand AMR may be used. Further, not only TS packets that transmit videocoded data but also audio or program information may be included in theTS packets to be inputted. In this case, TS packets including programinformation are obtained first, and then PIDs of TS packets thattransmit video and audio coded data are determined. After that, TSpackets are divided into video TS packets and audio TS packets based onPID values and then processed respectively.

FIG. 8 is a diagram showing the whole structure of a multiplex schemeconversion apparatus. The apparatus is composed of a data I/O unit 101and a conversion unit 103. The data I/O unit 101 inputs TS packet datato store the data in a memory 102 and outputs the MP4 file for whichmultiplex scheme is converted, and the conversion unit 103 that obtainsthe TS packet data from the memory 102, separates and analyzes the AUdata and the header of a PES packet, stores the analysis result in thememory 102, converts the separated AU data into an MP4 using theanalysis result, and outputs the conversion processing result to thememory 102. Here, the memory 102 includes a storage region of input andoutput data and a work region for temporally stores the conversionprocessing result or the AU data, and the sizes of the respectiveregions are changeable. Also, the restructure unit 103 is a CPU here,but it can be exclusive hardware. Operations of the conversion unit 103will be described in detail below.

FIG. 9 is a block diagram showing the structure of the conversion unit103. The conversion unit 103 includes a TS payload separation unit 201,a packet loss judgment unit 202, a PES separation analysis unit 203, anAU separation unit 204, a PES header analysis unit 205, a Box makingunit 206 and a connection unit 207. The TS payload separation unit 201separates the header of a TS packet and the payload and output them tothe packet loss judgment unit 202. The packet loss judgment unit 202judges a loss of a TS packet, performs error processing in the casewhere a packet loss is detected and outputs, to the PES separationanalysis unit, the payload data on which error processing has alreadybeen performed. The procedure of the judgment on whether or not a packetloss has occurred is as follows: (i) obtaining continuity_counter valuefrom the TS header; comparing the value with continuity_counter value ofthe TS packet inputted immediately before; and (iii) judging that apacket loss has occurred in the case where these values arediscontinuous. Here, the maximum value of continuity_counter is 15, andin the case where the continuity_counter value of a TS packet inputtedimmediately before is 15 and the continuity_counter value of current TSpacket is 0, it is judged that continuity_counter values are cycled andthere is no packet loss. In contrast, in the case where a packet loss isdetected, a NAL unit whose NAL unit type is 25 and whose payload isvacant is inserted, as information for notifying the presence of apacket loss, in the top of the payload data of current TS packetemploying a byte stream format. Note that the NAL unit type of the NALunit to be inserted can have an arbitrary value between 24 and 31inclusive on condition that the NAL unit type is not used for otherpurposes. Here, the NAL unit type of 0 and between 24 and 31 inclusiveare defined as areas that can be arbitrary used by an application in theAVC standard. Also, as the information for notifying the presence of apacket loss, a code pattern that does not appear in the AVC stream maybe inserted and the packet loss information may be managed separatelyfrom the stream. Next, the PES separation analysis unit detects thestarting position of the PES packet by searching the start code of thePES packet in the MPEG-4 AVC from the payload data of the TS packet intowhich the NAL unit for notifying the presence of a packet loss isinserted, the start code being between 0x000001EO to 0x000001EFinclusive. Here, in the case where a field indicating the size of a PESpacket is included in the header of the PES packet, the payload may beseparated using the size. After the PES packet is separated, the PESpacket is divided into the header and the payload, and the header datais outputted to the PES header analysis unit 205 and the payload data isoutputted to the AU separation unit 204. Note that the stuffing data inthe PES packet is not included in the payload data. In the AU separationunit 204, the boundary of AUs is detected from the payload data of thePES packet and the AU data is separated, and the separated AU data isoutputted to the Box making unit 206. Here, in the case where a NAL unitfor notifying the presence of a packet loss is inserted by the packetloss judgment unit, a NAL unit for notifying the presence of a packetloss is also included in the AU data to be outputted to the Box makingunit 206. The PES header analysis unit 205 judges whether the displaytime of the leading AU that starts in the PES payload is included in thedata of the inputted PES header, and when included, the display time ofAU is outputted to the Box generation unit 206. Next, the Box makingunit 206 makes ‘moov’ and ‘mdat’ including packet loss information basedon the inputted AU data and the display time in the PES header, andoutputs them to the connection unit 207. Note that the Box making unit206 may make ‘moof’ or ‘mfra’ in addition to ‘moov’ and ‘mdat’. Lastly,the connection unit 207 connects ‘moov’ with ‘mdat’ and outputs it asthe data of the MP4 file.

FIG. 10 is a flow chart showing the operation of the conversion unit103. First, judgment on whether a loss of TS packet occurred is made inthe step 301, and in the case where the occurrence of the packet loss isdetected, a NAL unit for notifying the presence of a packet loss isinserted in the position at which a packet loss has been detected.Consequently, the header and the payload of the PES packet are separatedfrom each other in the step 302, and the boundary of AUs is detectedfrom the payload of the separated PES packet and the AU data isseparated in the step 303. As shown in the Loop A of FIG. 10, theoperation from step 301 to step 303 are repeated until the data for 1 AUcan be separated. After the data for 1 AU is separated, in the step 304,the header of the PES packet is analyzed, judgment on whether thedisplay time of the leading AU that starts in the PES payload isincluded is made, and when included, the display time of the AU isobtained. Note that the processing of step 304 may be performed in theprocessing of Loop A. Next, in the step 305, the Box data in the MP4file is made based on the AU data separated in the Loop A and thedisplay time information obtained in the step 304. The processing ofLoop B is repeated until the processing of all the AU data included inthe TS packet to be inputted is completed, ‘moov’ and ‘mdat’ arecompleted at the time when the processing of all the AU data iscompleted, and in the step 306, ‘moov’ and ‘mdat’ are connected witheach other, and the MP4 file is outputted. Here, the Loop B includesprocessing of Loop A, step 304 and step 305. Note that ‘fytp’ is made,added to ‘moov’, and then outputted in the step 305 or added to ‘moov’in the step 306. Note that an MP4 file where fragments are used may beoutputted. In this case, in the step 305, at the time when ‘moof’ and‘mdat’ for one fragment are determined, ‘moof’ and ‘mdat’ are outputtedto finish Loop B, the outputted ‘moof’ and ‘mdat’ are sequentiallyconnected with each other in the step 306. In addition, at the time ofusing fragments, it is possible to store, in ‘moof’, the headerinformation of all the samples in a media track and also store, in‘moov’, the header information of the sample placed in the top of a‘trak’.

The structure and operations of the Box making unit 205 will bedescribed in detail. Note that the NAL unit included in the payload of aTS packet is simply described as a NAL unit from here, and it does notmean the NAL unit for notifying the presence of a packet loss. FIG. 11is a block diagram showing the structure of the Box making unit 205. TheBox making unit 205 includes an AU analysis unit 301, an errorinformation making unit 302, an AU format conversion unit 303, and a Boxdata determination unit 304. First, the AU analysis unit 301 analyzesthe AU data, obtains the information necessary for making a Boxcomposing a conventional MP4 file, and outputs the analysis result tothe Box data determination unit 304, such information being the displaytime of an AU, whether or not it is the AU IDR and the like. Here, theanalysis result outputted by the AU analysis unit 301 does not includethe size of the AU. Also, in the Box composing a conventional MP4 file,a Box indicating error information (loss information) or identificationinformation for indicating the error information in ‘mdat’ is notincluded.

The operation that is performed in the case where AU time information,an SPS or an PPS data that are referred to in decoding AU data is lostbecause of a packet loss will be described below. As to the AUs whosedecoding time or display time cannot be obtained from the headerinformation of the PES packet, the data of AU is analyzed and then thetime information is obtained. To be more specific, the parameterindicating the display order of pictures called Picture order count orSEI such as the Picture timing SEI and the Buffering period SEI areanalyzed. In the case where the information indicating that these piecesof time information are lost because of packet losses, the decoding timeor display time of each AU is calculated according to a predeterminedmethod. In the case where a SPS or a descriptor information transmittedby TS packets having a PID different from coded media data indicate thatthe frame rate is fixed, the decoding time or display time may bedetermined based on the frame rate. Note that the descriptor issupplementary information of the coded media data that is described inthe MPEG-2 system standard. Next, in the MP4, SPSs and PPSs are storedin a sample entry in the sample description box (‘stsd’) or a ‘trak’ fora parameter set. Further, (i) a sample in the ‘trak’ of the MPEG-4 AVCand (ii-i) each sample entry that stores an SPS and a PPS that arereferred to by the sample or (ii-ii) each sample of the ‘trak’ in aparameter set are associated with each other one by one. Therefore, inthe case where the data of an SPS or a PPS that is referred to indecoding a sample is lost, the same SPS or the PPS used in decoding theimmediately-before-sample is referred to instead. Note that the defaultSPS or PPS that is previously set may be used.

In the error information making unit 302, a NAL unit for notifying thepresence of a packet loss is searched among AU data, and in the casewhere such NAL unit is detected, a Box data indicating error informationis made. Further, the NAL unit data to be stored in ‘mdat’ is determinedbased on the presence or absence of the packet loss in the data of theNAL unit composing AUs, and it is determined that the size of AU datacomposed of NAL unit data is stored in ‘mdat’ as sample data. Afterthat, those pieces of information are outputted to the Box datadetermination unit 304 together with the Box data indicating errorinformation. In the AU format conversion unit 303, in the errorinformation making unit, the data of the NAL unit in the AU to be storedas sample data is converted from the byte stream format to the NAL sizeformat, and the resulting AU data is outputted to the Box determinationunit 304. Here, the converted AU data outputted in the Box determinationunit 304 becomes sample data in the MP4.

FIG. 12 is a flow chart showing the operation of the Box making unit205. First, in the step 401, the information necessary for making Boxescomposing a conventional MP4 file is obtained from the inputted AU data.In the step 402, Box data indicating error information is made bydetecting a NAL unit for notifying the presence of a packet loss from AUdata. Further, in the step 403, the NAL unit data to be stored as sampledata in the MP4 file is determined. In the step 404, the NAL unit datawhich is determined as sample data is stored in the step 403 isconverted to a NAL size format, and the converted NAL unit data composesa sample data is made. Consequently, in the step 405, the ‘moov’ dataincluding the Box indicating error information is made. In the step 406,‘mdat’ data that stores sample data made in the step 404 is made. FIG.13 is a flow chart showing the operation of step 405. In the step 404,Boxes composing a conventional MP4 file is made in the step 501 first,and next, a Box for storing error information is made in the step 502.

Next, in the step 402, the procedure taken at the time of making a Boxdata showing the error information will be described. FIG. 14 is a flowchart showing the operation of step 402. Here, a technical word ofsample is used for describing the Box data making method in the MP4file, and one sample means 1 AU. First, in the step 601, search forjudging whether a NAL unit for packet loss notification is included inAU data is performed. In the search, the starting position of a NAL unitis detected by checking starting from the leading data of an AU in ordera start code prefix (0x000001, 3 bytes) of a byte stream format in abyte align position. Consequently, the NAL unit type of the detected NALunit is obtained, and in the case where the resulting value of the NALunit type is a predetermined value, the NAL unit is judged as a NAL unitfor notifying the presence of a packet loss. After the NAL unit types ofAUs are checked up to the last one in order, judgment on whether or nota NAL unit for notifying the presence of a packet loss is detected in asample is made in the step 602. The processing of step 603 is performedin the case where a NAL unit for notifying the presence of a packet lossis detected, in contrast, in the case where a NAL unit for notifying thepresence of a packet loss is not detected, the Box data makingprocessing is completed. In the step 603, the sample numbers of samplesare added to a table showing a list of samples including an error.Operations performed in the step 601 to step 603 will be described belowwith reference to concrete examples as FIG. 15A to 15D. FIG. 15A is thediagram of sample data in the coded data of AVC, in the diagram, thesample data are placed in a decoding order and, the numbers in thefigure show sample numbers respectively. In this example, a NAL unit fornotifying the presence of a packet loss is detected in the NAL unit datain the samples whose sample numbers are 7, 9 and 15. As to samples whosesample numbers are 1 to 15, the procedure taken at the time of making aBox indicating error information will be described below. Note that thename of the Box indicating error information is Error sample box. First,as a NAL unit for notifying the presence of a packet loss is detected inthe step 601 as to first to sixth samples, it is judged in the step 602that a NAL unit for notifying the presence of a packet loss is presentin a sample, and the processing is completed without updating the tabledata of Error sample box. As a NAL unit for notifying a packet loss isdetected in the seventh sample in the step 601, it is judged in the step602 that the table data of Error sample box is updated, and the updateprocessing is performed in the step 603. In the step 603, a samplenumber 7 is stored by adding the entry of a table in order to indicatethat a packet loss is included in the seventh sample. The sameprocessing is performed on the following samples, and the sample number9 and the sample number 15 are added to the table. As a result, numbersof 7, 9 and 15 are stored, as the numbers of samples including a packetloss, in the table of Error sample box as shown in FIG. 15B. FIG. 15C isa syntax example of Error sample box, and error_sample_number shows thesample number of a sample whose NAL unit data is partly lost because ofa packet loss. The Error sample box is placed immediately below ‘stbl’in trak in ‘moov’, and it is placed immediately below a track fragmentrun box (‘traf’) in ‘moof’, but it may be placed immediately below oneof the other Boxes. Here, an Error samle Bbx is made only in the casewhere a sample whose data is lost is present in a media track, but anError sample box whose entry is vacant may be made even in the casewhere a sample whose data is lost is not present. Note that, at the timeof placing Error sample boxes in traf, these samples are given anerror_sample_number obtained by counting these samples starting from theleading sample whose sample number is 1, the leading sample beingincluded in ‘traf’. For example, in the case indicating that ten samplesare included in ‘traf’ and that the data of third and fifth samples arelost because of packet losses, two entries are made, one of which beingassigned 3 and the other one of which being assigned 5 as itserror_sample_number. Also, it is possible to add a flag indicating thatdata in a sample is lost because of a packet loss to tr_flags in a trackfragment run box (‘trun’), the tr_flags being a field indicating flaginformation concerning the header information of each sample.

Further, whether or not the data in a sample is lost because of a packetloss is shown in the above explanation, but whether or not the data islost in a specific NAL unit of a sample may be shown. For example, inthe MPEG-4 AVC, whether or not the data is lost in the NAL unit data ofa slice, an SPS or a PPS is shown. This is because a correct decodingresult can by obtained under the condition that slices and parametersets to be referred to by the slice are completely available. To be morespecific, as to a NAL unit of a parameter set, whether or not the dataof an SPS or a PPS that is necessary for decoding slice data of a sampleis lost may be shown. Also, as a syntax shown in FIG. 15D, theinformation for identifying a NAL unit whose data is partly lost may bestored. In the figure, error_nal_unit_number is a number for identifyinga NAL unit whose data is lost, and in the case where the second and thefourth NAL units in a sample are lost, two entries whoseerror_nal_unit_numbers are 2 and 4 respectively are assigned. In theentries, a NAL unit type may be shown. Further, an error_nal_unit_numberis used for identifying a video packet in the MPEG-4 visual or a slicein H. 263.

Also, in the case where the information on not only a packet loss butalso a bit error may be obtained, it is possible to store theinformation for identifying a sample or a NAL unit whose sample dataincludes a bit error.

Next, operations, which are performed in the steps 403 and 404respectively, of determining a NAL unit data to be stored as a sampledata in the MP4 file and converting the storage format of the NAL unitto the NAL size format will be described below with reference to theflow chart of FIG. 16. First, in the step 701, valid data in each NALunit in an AU is determined and separated. Here, in the case where a NALunit for notifying the presence of a packet loss is detected, such validdata in the NAL unit whose starting position has already been detectedimmediately before is determined as the data up to immediately beforethe NAL unit for notifying the presence of a packet loss. In this case,the data of NAL units that follow the detected NAL unit for notifyingthe presence of a packet loss becomes invalid, in other words, thepresence of NAL units placed before the next NAL unit whose startingposition is to be detected becomes invalid. This is because there is acase where the NAL unit to which the data including an invalid partcannot be specified, or because, even if specified, the size of the lostdata is unknown and the data cannot be used in decoding. In the casewhere NAL unit for notifying the presence of a packet loss is notdetected, all the NAL unit data is regarded as valid data. Note thatzero_byte and a start code prefix are not included in the NAL unit dataseparated as valid data. Next, the size of the valid data separated inthe step 701 is obtained in the step 702. In the step 703, the NAL unitdata including the valid data is converted to the NAL size format basedon the size obtained in the step 702. Here, the size of the fieldindicating the size of the NAL unit is variable within the range of 1 to4 in the NAL size format, but the field size is regarded as givenpreviously. Next, whether the NAL unit data is lost or not is judged inthe step 704, and in the case where it is lost, a NAL unit for errornotification is inserted as a NAL size format. The processing from theabove step 701 to step 705 (Loop A) is repeated until the processing ofthe last NAL unit in a sample is completed. Note that the size of asample is determined at the time when the size of the last NAL unit inthe sample is determined.

Also, whether the data of a sample is lost or not is shown only in theBox indicating error information stored in ‘moov’ or ‘moof’, and a NALunit for notifying the presence of a packet loss may not be insertedinto the sample data to be stored in ‘mdat’. At this time, the steps 704and 705 become unnecessary.

Also, it is possible to store all the NAL unit data included in thepayload of a TS packet may be stored without throwing away the data thathas been judged as invalid. Further, in the case where the size of thedata that is lost because of a packet loss can be specified, a specificNAL unit (called a NAL unit for complementing a packet loss from here)for stuffing the lost part may be inserted. A NAL unit type that is inan unspecified region and is assigned, as its NAL unit type value, anumber different from the one for notifying the presence of a packetloss is used in the NAL unit for complementing a packet loss. Also, theerror information may be shown only in the NAL unit for errornotification, and an Error sample box may not be made.

Also, in the case where the packet loss information in the NAL unit datain an AU is provided separately from the NAL unit for notifying thepresence of a packet loss, the valid data in the NAL unit is determinedbased on the information separately shown in the step 701.

FIG. 17 and FIG. 19 respectively show a concrete example of determiningvalid NAL unit data and converting it to the NAL size format. In eachdiagram, SCP, Nal_len, ERR_NALU is a start code prefix in a byte streamformat, the size of a field showing the size of a NAL unit in the NALsize format, and a NAL unit for error notification respectively. FIG. 17is an example where NAL unit data because of a packet loss occurswithout crossing a NALU boundary. FIG. 17A shows before-conversion AUdata. The AU is composed of four NAL units of NALU #1 to NALU #4, and apacket loss occurs in the third NAL unit. The NALU #3 is basicallycomposed of data in the regions shown in the figure as “receive 1”,“lost” and “receive 2”, but ERR_NALU is inserted immediately after thedata of the “receive 1” because a “lost” region is lost because of apacket loss. At this time, as only the data of “receive 1” part isjudged as valid data in NALU #3, only the data of “receive 1” part isstored as the sample data of ‘mdat’. As the data size of the “receive 1”part is size3_(—)1, the NAL_len field value is size3_(—)1 at the timewhen converting the data of NALU #3 to the NAL size format. Next,immediately after the NALU #3, ERR_NALU of the NAL size format isinserted. FIG. 17B shows the structure of AU data outputted afterconversion. Note that the data structure of AU at the time of storingall the NAL unit data included in the payload of a TS packet withoutinserting a NAL unit for error notification is shown as FIG. 17C, andthe size of NALU #3 is the total of size3_(—)1 and size3_(—)2 (the sizeof “receive 2”).

FIG. 18 is an example where NAL unit data is lost because of a packetloss crossing the boundary of NAL units. FIG. 18A showsbefore-conversion AU data. The AU is composed of four NAL units of NALU#1 to NALU #4, and the data is lost crossing a NAL unit, morespecifically, from the ending part of NALU #2 to the leading part ofNALU #3 (the data of the region shown as “lost” in the figure). At thattime, the data of “receive 1” part in the figure becomes valid in NALU#2. Next, as to NALU #3, the data immediately after ERROR_NALU to thedata up to the starting position of NALU #4 becomes invalid because thestarting position of a NAL unit cannot be detected. As a result, thedata of NALU #3 is not stored in the sample data. In other words, thedata of “receive 2” part in the figure becomes invalid. FIG. 18B showsthe structure of AU data outputted after conversion.

FIG. 19 is an example where NAL unit data is lost because of a packetloss crossing the boundary of AUs. FIG. 19A shows before-conversion AUdata. Three AU data from AU #1 to AU #3 are composed of NALU #1 to NALU#4, NALU #5 to NALU #8 and NALU #9 to NALU #12 respectively. Here, datafrom the middle of NALU #4 to the leading part of NALU #9 is lostcrossing AU #2 (the data of a region shown as “lost” in the figure). Thevalid data in AU #1 to AU #3 is described below. In AU #1, all the datafrom NALU #1 to NALU #3 and the data of nal4_size obtained in NALU #4become valid. As all the AU #2 data is lost, there is no valid data.Therefore, in the MP4 file, a sample for storing AU #3 will be storednext to the sample for storing AU #1. At this time, as a reproductionduration of a sample for storing AU #1, the total of reproductionduration of AU #1 and AU #2 is stored. Lastly, in AU #3, the data fromNALU #10 to NALU #12 become valid. Here, ERR_NALU may be inserted intothe leading part of NALU #10 in the case where it is possible todetermine that NALU #10 is not the leading NAL unit in AU #3. Forexample, as it is prescribed in the MPEG-2 system standard that a NALunit for determining the boundary of AUs called Access unit delimitershould be inserted into the top of an AU, it is possible to judge thatthe leading NAL unit of AU #3 is lost unless NALU #10 is the NAL unit ofAccess unit delimiter. FIG. 19B shows the structure of AU data outputtedafter conversion. FIG. 19C shows the structural example of AU dataoutputted at the time of using a NAL unit for complementing a packetloss. The data of AU #1 and AU #3 are the same as the data shown as FIG.19B, but in this example, the data of AU #2 is also outputted. Theoutput data of AU #2 includes only the NAL unit for complementing apacket loss, and the size of the NAL unit for complementing a packetloss can be estimated based on the information on whether it is thecoded bit rate of the MPEG-4 AVC data and a fixed frame rate. In thisway, inserting a NAL unit for complementing a packet loss makes itpossible to store the sample data in the MP4 by holding the originalfixed bit rate as much as possible also in the case where the data islost because of a packet loss while video data is being coded in a fixedbit rate. Also, in the case where the stream has a fixed frame rate,making a sample composed of NAL units for complementing a packet lossmakes the reproduction duration of a sample become the same, and as aresult, it becomes possible to reduce the size of a table that storesthe reproduction duration of the sample.

Also, in the case where the slice data of an AU is lost, the same dataas the data of the AU that is placed immediately before in a decoding ordisplaying order is stored. In other words, it is possible toreconstruct and record the AU data by performing error concealmentprocessing on a slice-by-slice basis. Further, it is also possible topresent the information indicating that the AU data is reconstructed.

Note that it is possible to store the information on a packet loss byusing a Supplemental enhancement information (SEI) message. For example,the SEI message for storing user data can be used.

Note that error information may be stored in the payload or the Errorsample box of the NAL unit for error notification, such errorinformation being the rate of data that is lost because of a packetloss, the rate of slices whose data are lost in an AU or the frequencyof bit errors. Also, in the case of describing the types and the sizesof NAL units for error notification, or error information in the payloadof the NAL unit, a Box for indicating the type of error information tobe described may be stored in ‘moov’ or ‘moof’.

Note that each of the above-mentioned operations becomes applicable alsoin the case of storing AUs as one sample by using forme_count field inthe sample entry in a sample description box (‘stsd’).

Further, the method of making a Box for error notification or insertinga NAL unit for error notification in sample data is applicable also inthe case of converting, to an MP4 file, the coded media data multiplexedby a Real-time transmission protocol (RTP) packet. In the RTP packet,judgment on whether a packet loss occurred or not can be made bychecking the continuity of sequence numbers added to the headers of theRTP packets.

Also, in this embodiment, error information such as an Error sample boxis added to each ‘trak’, but the unit for adding error information isnot limited to this. For example, error information can be added on anMP4 file-by-MP4 file basis. In this case, error information can beplaced immediately below ‘moov’. Also, at the time of using a fragment,it may be placed in the ‘moof’ of the last fragment or in the Box of‘mfra’ and so on. This enables notifying a user of information such as aloss rate (lost data/all data) of sample data and the rate of durationto the whole reproduction duration during which audio and video can bereproduced concurrently.

In addition, it is possible to have items for which a predeterminedreproduction segment is specified are possessed in a form of a list, andadd error information to each item in the play list where thereproduction order of these items are described. In the case of audiodata, a song is regarded as an item and songs selected by a user aredescribed in a selection order in this play list. Here, respectivereproduction segment may be different segment in the same file, orpredetermined areas of different files. In this case, referring to theplay list at the time of reproduction enables obtaining information suchas a loss rate of sample data for each item.

Second Embodiment

The multiplex scheme conversion apparatus concerning the secondembodiment of the present invention will be described with reference toFIG. 20 and FIG. 21. The structure of the multiplex scheme conversionapparatus is approximately the same as the multiplex scheme conversionapparatus in the first embodiment, but it differs in its operationperformed by the Box data determination unit 304 when making ‘stss’ thatis the table of a Sync sample. Therefore, only the ‘stss’ making methodwill be described below.

In the multiplex scheme conversion apparatus in the first embodiment, anIDR AU is a Sync sample regardless of whether data in a sample is lostor not. However, as the Sync sample becomes the sample from whichdecoding is started at the time of random access processing, the Syncsample that cannot be correctly decoded may cause a problem that samplesfollowing the Sync sample in a sequence cannot be decoded correctly.Therefore, only an IDR AU whose data is not lost is made to be a Syncsample in the multiplex scheme conversion apparatus of this embodiment.This makes it possible to obtain at least a non-turbulent Sync sample atthe time of starting decoding from the Sync sample.

FIG. 20 is a flow chart showing the operation at the time of making‘stss’ in the Box data determination unit 304. First, in the step 801,judgment on whether the NAL unit of an IDR slice is included in a sampledata or not is made. Here, whether it is the NAL unit of the IDR sliceor not is identified by its NAL unit type. Current processing iscompleted in the case where a NAL unit of the IDR slice is not included,but in the case where such a NAL unit is included, the processing ofstep 802 is performed. Also, in the step 802, judgment on whether a NALunit whose data is lost because of a packet loss is included in a sampledata is made based on the error information for each sample obtained inthe error information making unit 302. Current processing is completedin the case where sample data is judged as lost, but in the case wheresample data is judged as not lost, the sample number of the sample isadded to the table of ‘stss’ indicating a list of Sync samples in thestep 803. In this way, ‘stss’ is made by repeating processing of step801 to step 803 until the last sample in a ‘trak’ is processed. Notethat, instead of using, as Sync samples, the samples that include theNAL unit of an IDR slice and whose data is not lost, it is possible todetermine Sync samples so that the differential value of reproductionstarting time between Sync samples become constant as much as possible.Also, it is possible to determine, as Sync samples, samples whose dataare not lost in the slice, the SPS or the NAL unit of a PPS in an AUincluding an IDR slice. To be more specific, it is possible todetermine, as Sync samples, only in the case where the data of an SPS ora PPS that is necessary for decoding slice data of a sample can beobtained from the current sample or the samples with an earlier decodingtime.

FIG. 21 shows a concrete example at the time of making ‘stss’ by theoperation shown in the flow chart as FIG. 20. FIG. 21A shows samplesaligned in the ascending order of sample numbers. The samples whosesample numbers are 1, 11, 21 and 31 include a NAL unit of an IDR slice,but the data in the sample data of the twenty-first sample is lost. Atthat time, the first, eleventh, twenty-first and thirty-first samplesare considered as candidate Sync samples in the step 801, but in thestep 802, it is determined that the twenty-first sample is not made tobe a Sync sample. As a result, in the step 803, the first, eleventh andthirty-first samples are added to the table of ‘stss’ (FIG. 21B).

Note that samples to be recorded in ‘mfra’ as randomly-accessiblesamples can be determined in a similar way.

Note that making ‘stss’ using the above-mentioned method and makingError sample box can be selected independently. For example, it ispossible to make ‘stss’ using the above-mentioned method but not to makeError sample box.

Third Embodiment

The multiplex scheme conversion apparatus concerning the thirdembodiment of the present invention will be described with reference toFIG. 22 to FIG. 24. The structure of the multiplex scheme conversionapparatus is approximately the same as the structure of the multiplexscheme conversion apparatus in the first embodiment, but it differs inits operation of step 402 in FIG. 12 performed by the error informationmaking unit 302 when making a Box indicating error information of asample. Therefore, only this operation will be described below. In themultiplex scheme conversion apparatus, a Box indicating whether up towhich sample, among samples that follows the Sync sample, can becorrectly decoded is made at the time when decoding is started from theSync sample in the multiplex scheme conversion apparatus. At the time ofreproducing an MP4 file made in the multiplex scheme conversionapparatus referring to this Box makes it possible to determine thesample to be displayed after the samples guaranteed as correctlydecodable are specified.

FIG. 22 is a flow chart showing the operation performed at the time ofmaking a Box indicating error information. Here, err_free_flag is a flagindicating that data loss has not occurred in samples placed between thesample from which a sequence is started to the current sample, and itsinitial value is 0. In the step 901, judgment on whether or not the NALunit of an IDR slice is included in a sample is made, and if included,err_free_flag is set to 1 in the step 902. In the step 903, whethererr_free_flag is 1 or not is judged as follows: the processing of step904 is performed in the case where the value is 1; or in the case wherethe value is 0, the following steps are skipped and the processing ofthe next sample is started. In the step 904, judgment on whether or notthe data of the NAL unit that composes a sample is lost is made.Furthermore, it is possible to judge that data is not lost in the step904 in the case where NAL unit data of an SPS or a PPS is not lost. Inthe case where a NAL unit whose data is lost is included, the samplenumber of the immediately-before sample is added to the table showingthe list of last samples of the respectively corresponding error freeareas in the step 905. Here, “error-free area” means the area of samplesthat are placed in a decoding order and guaranteed as free from a dataloss. At the time of performing a random access, decoding is startedfrom a Sync sample, and further, the starting sample of an error-freearea is a Sync sample because a Sync sample includes a NAL unit of anIDR slice. Note that, in the case where the data of a sample including aNAL unit of an IDR slice is lost, an error-free area is not defined forthe sample. Consequently, step 905 is skipped. In this way, theprocessing from step 901 to step 905 is repeated until the processing ofthe last sample in a ‘trak’ is completed. Here, guaranteeing that a dataloss has not occurred indicates that an error is not detected judgingfrom the continuity_counter value of a TS packet, but does not alwaysguarantee that no data loss has occurred. Note that, in the case wherepacket losses can be detected more correctly based on packet lossinformation except continuity_counter, an error may be detected usingthe information.

Note that it is also possible to use information except the last sample,for example, by indicating the number of samples included in anerror-free area as long as it is possible to indicate the error-freearea.

FIG. 23 shows a concrete example at the time of making a Box for storingerror information using the operation shown in the flow chart as FIG.22. FIG. 23A shows two continuous sequences, each of which beingcomposed of ten pieces of samples. In the sequence #1, starting from thefirst sample, the data of the seventh and ninth samples are lost, and inthe sequence #2, starting from the eleventh sample, the data of thesixteenth sample is lost. The procedure taken at the time of determiningan error-free area will be described below. First, the err_free_flag isset at 1 for the first sample. As it is judged that the data of samplesup to the sixth sample are not lost in the step 904, processing of steps905 and 906 are skipped and the value of err_free_flag is held at 1.Next, as it is judged that the sample data of the seventh sample is lostin the step 904, an entry is added to the table for storing the lastsamples of the respectively corresponding error-free areas in the step905, and the sample number indicating the sixth sample that is theimmediately-before sample is stored. The err_free_flag values of thesamples between seventh to tenth samples inclusive remains 0.Consequently, the err_free_flag value of the eleventh sample is set at 1again, and likewise the case of sequence #1, the fifteenth sample isadded to the table as the last sample of the corresponding error-freearea. FIG. 23B shows a Sync sample table. FIG. 23B shows the table ofthe last sample of an error-free area. The two tables of FIG. 23B andFIG. 23C show that samples whose sample numbers are one to six areincluded in the error-free area in the sequence #1, and that sampleswhose sample numbers are eleven to fifteen are included in theerror-free area in the sequence #2. FIG. 23D is an example syntax ofError free end sample box that is a Box indicating a list of lastsamples of the respectively corresponding error-free areas, andend_sample_number shows the sample numbers of the last samples of therespectively corresponding error-free areas. An Error free end samplebox is placed immediately below ‘stbl’ in the ‘trak’ in ‘moov’, orimmediately below a track fragment run box (‘traf’) in ‘moof’, but itmay be placed immediately below another Box.

Note that whether or not a NAL unit of an IDR slice is included in asample is judged and an err_free_flag is set in the step 901, but in thecase where the sample including the NAL unit of an IDR slice is notalways made to be a Sync sample, it is impossible to specify theerror-free area using the Sync sample list shown in ‘stss’ and the Errorfree end sample box. Therefore, it is possible to judge whether or notthe sample is a Sync sample and, if it is the Sync sample, determinesetting an err_free_flag to the Sync sample. Also, it is possible toshow not the last sample of the respectively corresponding error-freeareas but the leading samples. For example, in the case where all thedata of the sixth to tenth samples of the sequence #1 are completelyavailable in FIG. 23A, it is possible to show that the sixth sample tothe eleventh sample that is the leading sample of the sequence 2 areincluded in the error-free area by assigning 6 to the leading sample ofthe error-free area.

Note that, in the MP4, it is possible to specify the Gradual decoderrefresh point, other than a Sync sample, as a randomly-accessiblesample. The Gradual decoder refresh point indicates the sample havingthe following features: It cannot be decoded correctly; It enablesdecoding the following samples correctly under the condition that thespecified number of following samples are decoded in a decoding order;and It enables indicating the sample from which decoding is started inthe case where there is a need for starting decoding from the sampleplaced earlier by the specified number in the decoding order in theinterest of decoding a sample correctly. For example, when decoding isstarted from the “M”th sample in the decoding order, in the case wheresamples that can be decoded correctly are the “N”th sample (where N>M)and the following samples, the information indicating the differencebetween N and M is shown in the MP4 file. Therefore, it is possible touse the sample corresponding to the Gradual decoder refresh point as theleading sample of the corresponding error-free area.

Note that it is possible to concurrently indicate the leading sample andthe last sample of an error-free area so that the error-free area can bespecified without being associated with a Sync sample.

The list of last samples of the respectively corresponding error-freeareas is shown in the above explanation, but it is possible to indicatewhether or not a sample whose data is lost is included in a sequence.The case will be described below with reference to FIG. 24. FIG. 24Ashows six continuous sequences, each of which being composed of tensamples. Here, all the sample data are completely available in the foursequences of #1, #3, #4 and #6, but in contrast, a sample whose data islost is included in the sequences of #2 and #5. FIG. 24C is a list ofleading samples in the corresponding sequences where sample data in asequence are completely available. FIG. 24B is a list of Sync samples inthe case where leading samples of the respectively correspondingsequences are made to be Sync samples. With reference to the two tablesof FIG. 24B and FIG. 24C, it is possible to know that the sequenceswhere a data loss has occurred starts from the Sync samples whose samplenumbers are 11 and 41 and that the sequences where a data loss has notoccurred starts from the Sync samples whose sample numbers are 1, 21, 31and 51. Note that in the case where samples including a NAL unit of anIDR slice are not always made to be a Sync sample, it is also possibleto indicate whether or not a data loss has occurred in the sample databetween the two continuous Sync samples. FIG. 24D is an example syntaxof the Error free sequence box indicating the list of the leading sampleof the sequence where a data loss does not occur. Here,sequence_start_sample_number in the syntax shows the sample number ofthe leading sample of the sequence. Note that it is also possible toindicate an index such as good, medium and bad as a parameter indicatingthe quality of the reproduction video obtained at the time ofreproducing a sequence depending on the data loss rate in a sequence inaddition to the leading sample numbers of a sequence. For example, it ispossible to determine the quality depending on the data loss rate, theappearance frequency of intra-coded AUs and the rate ofintra-macroblocks. Further, it is possible to store the informationindicating the samples that can be decoded correctly in the sequence. Tobe more specifically, indicating the reference relationship betweensamples and whether or not data is lost in the NAL units of slice data,SPSs or PPSs in each sample makes it possible to know whether or not thedata of the reference-destination sample is lost at the time of decodingthe sample. Here, Error free sequence box is placed immediately below‘stbl’ in the ‘trak’ in ‘moov’, and immediately below a track fragmentrun box (‘traf’) in ‘moof’, but it may be placed immediately belowanother Box.

Also, it is possible to use, in combination, Error sample boxes, Errorfree end sample boxes and Error free sequence boxes that have alreadybeen described in the first to third embodiments. For example, using anError sample box in combination with an Error free end sample box makesit possible to identify the sample whose data is lost, among samplesoutside the error-free area, and determine samples to be decoded.

Also, it is possible to determine the reproduction quality of a ‘trak’based on the rate of samples that can be decoded correctly or sampleswhose data is lost and place a Box where the information indicating thereproduction quality is stored at the position immediately below ‘trak’,or to place, at the position immediately below ‘moov’, a Box where theinformation indicating the reproduction quality is stored at the time ofconcurrently reproducing a ‘trak’ of audio, video or text data.

Note that, in the above-mentioned respective embodiments, it is possibleto perform processing of the following parts separately: the part to beconverted from a byte stream format to a NAL size format and the part towhich the information indicating a data loss is added. For example, itis possible to add data loss information to the NAL unit datatransmitted in a form of a NAL size format or a format used in videodistribution on the Internet such as Real time transmission protocol(RTP).

Also, in the case of multiplexing coded data of the MPEG-4 AVC, AUformat conversion unit 303 can be used. Note that it is possible toconvert the format of the coded data into a byte stream format in orderto simplify the coding processing of the MPEG-4 AVC, and for example, itis possible to apply the processing of the AU format conversion unit 303at the time of multiplexing the coded data outputted from the codingunit using a multiplex scheme such as a TS and an MP4 whose storageformat of coded data is different. Here, processing for complementingdata loss can be omitted because an error is not present in a stream atthe time of coding. Note that the byte stream format is converted to theNAL size format in the AU format conversion unit 303, but it is possibleto convert the NAL size format to the byte stream format at the timewhen the format of the output data at the time of coding is convertedinto the NAL size format.

Fourth Embodiment

The demultiplexer concerning the fourth embodiment of the presentinvention will be described with reference to FIG. 25 to FIG. 27. Thedemultiplexer inputs the MP4 file made by the multiplex schemeconversion apparatus in the first to third embodiments, demultiplexes,decodes and displays the AU data. This demultiplexer determines decodingand displaying operations by referring to error information shown in anError sample box and the like. Therefore, it can determine the operationof skipping to the next sequence and the like based on file-format-levelinformation in the case where a lot of samples that cannot be decodedcorrectly are included in a sequence, and as a result, it can preventthe deterioration of reproduction quality caused by decoding anddisplaying samples that cannot be decoded correctly. Also, it can obtainthe same output result also in the case where a decode unit or a displayunit that performs different error concealment processing by determiningdecoding and displaying operations based on the file-format-levelinformation. Note that, in the case where a data loss in a sample isshown in the MP4 file to be inputted in the demultiplexer, a Boxindicating error information (loss information) is added. The operationperformed at the time of reproducing a ‘trak’ of the MPEG-4 AVC will bedescribed below. A ‘trak’ to be reproduced is not limited to the MPEG-4AVC, it may be coded video data such as H. 263 or the MPEG-4 visual,coded audio data such as the MPEG-4 AAC or AMR or the coded data thatconcurrently reproduces a ‘trak’ composed of video and audio data.

FIG. 25 is a block diagram showing the structure of the demultiplexer.The demultiplexer includes a header demultiplex unit 401, a headermemory 402, a ‘mdat’ memory 403, a sample obtainment unit 404, an errorinformation analysis unit 405, and a decoding and reproducing unit 406,and it starts reproduction processing triggered by reproduction startingtime to be inputted from outside. The reproduction starting time becomeszero second at the time of starting reproduction processing from the topof a file, or it becomes ten seconds at the time of startingreproduction processing from the position ten seconds later from thetop. Note that there is a case where a Sync sample having reproductionstarting time that matches the inputted reproduction starting time isnot present at the time of starting decoding and reproduction from aSync sample. In this case, decoding and reproduction is started from oneof the immediately-before Sync sample and the immediately-after Syncsample that has reproduction starting time nearest to the inputtedreproduction starting time. Note that it is also possible to startreproduction from not a Sync sample but the sample shown as a Gradualdecoder refresh point.

The header separation unit 401 separates a header part including ‘moov’and ‘moof’ and a data part composed of ‘mdat’ from the MP4 file, outputsthe data of the header part to a header memory 402 and outputs the dataof ‘mdat’ to a ‘mdat’ memory 403. The sample obtainment unit 404determines a Sync sample from which decoding is started based on thereproduction starting time information inputted from outside. As to thefollowing samples, the sample number of the Sync sample is outputted toan error information analysis unit 405 at the time of obtaining theheader information of the Sync sample. The error information analysisunit 405 obtains and analyzes error information in the sequence thatstarts from the Sync sample number inputted by the sample obtainmentunit 404 from an Error sample box, an Error free end sample box or anError free sequence box, determines the sample to be decoded in asequence, and after that, outputs the list of the samples to be decodedto the sample obtainment unit 404. Also, the error information analysisunit 405 outputs the obtained loss information to the loss informationnotification unit 406C of the decode and display unit 406. Also, in thecase where data loss information for each NAL unit of a slice can beobtained, the information indicating a slice to be decoded in a samplemay be outputted. Note that samples to be decoded may be determined onthe basis of predetermined number of samples instead of being determinedon a Sync sample-by-Sync sample basis. The sample obtainment unit 404analyzes ‘moov’ or ‘moof’, obtains the data and the display time of thesamples to be decoded, the samples being determined by the errorinformation analysis unit, and outputs them to the decode and displayunit 406. Here, ‘moov’ or “moof” are obtained from the header memory 402and the data of samples are obtained from the ‘mdat’ memory 403. Notethat, in the case where decoding time is different from display time,both of them may be outputted to the decode and display unit 406.

Lastly, the decode and display unit 406 includes a decode unit 406 a, asample display unit 406 b and a loss information notification unit 406c. The decode unit 406 a decodes sample data. The sample display unit406 b displays the decoded sample data according to their display time.The loss information notification unit 406 c notifies a user of theinformation such as the loss rate (lost data/all data) of sample databased on the loss information, and makes an inquiry as to whether theMP4 file should be reproduced or not. Also, on receiving a reproductionpermission signal in the case where a user permitted reproduction basedon the information such as the loss information of sample data, thesample obtainment unit 404 outputs the reproduction indication signal tothe sample obtainment unit 404. Note that the error information analysisunit 405 may not determine samples to be decoded but output the errorinformation, to the decode and display unit 406, such as the informationof samples whose data are lost or an error-free area including suchsamples. After that, the decode and display unit 406 may determinesamples or slice data to be decoded based on the inputted errorinformation or the information shown by the NAL unit for errornotification inserted into the sample data. At this time, a list ofdecoded samples is not outputted to the sample obtainment unit 404.

Next, the operation for reproducing the MP4 file performed by thedemultiplexer structured like above will be described. FIG. 26 is a flowchart showing the operation performed at the time of reproducing the MP4file.

The loss information notification unit 406 c notifies a user of the MP4file or loss information for each ‘trak’ that are obtained from theheader information (‘moov’) by the error information analysis unit 405and inquires the user of whether the user should reproduce the MP4 fileor not (step 1001). Next, whether or not the user indicated reproductionin reply to this inquiry is judged (step 1002). In the case where theindication is that reproduction should be performed (Yes in the step1002) as a result of the inquiry, the loss information notification unit406 c orders reproduction by outputting a reproduction indication signalto the sample obtainment unit 404 (step 1004). It starts reproductionfrom the inputted reproduction starting time, decodes samples until thelast sample is decoded or the user indicates the completion of thereproduction (step 1005). On the other hand, in the case where theindication is that reproduction should not be performed (No in the step1002) as a result of the inquiry, the user is notified that the MP4 fileshould not be reproduced (step 1003).

Next, the procedure performed at the time of reproducing samples in thestep 1005 will be described.

FIG. 27 is a flow chart showing the operation performed in the step1005. First, a Sync sample from which decoding is started is determinedbased on the inputted reproduction starting time in the step 1101. Afterthe processing of step 1101 is completed, the processing of Loop B isperformed. The Loop B indicates repeating the processing of step 1102 to1106, and it is repeated every time a new Sync sample is decoded indecoding. In the step 1102, error information as to whether data is lostor not is obtained in the samples between (i) an Error sample box, anError free end sample box or an Error free sequence box and (ii) thenext Sync sample. In the step 1103, samples to be decoded in thesequence are determined based on the error information obtained in thestep 1102, and a list of sample numbers of samples to be decoded isstored in a table. After the step 1103 is completed, samples to bedecoded based on the table made in the step 1103 are decoded anddisplayed in order in the processing of Loop A. The Loop A is composedof the processing from step 1104 to step 1106, and it is repeated untilthe processing of the last sample to be decoded at the time ofreproduction is completed. In the step 1104, sample data and sampledisplay time are obtained by analyzing ‘moov’ or ‘moof’. In the step1105, sample data obtained in the step 1104 are decoded, and in the step1106, sample data decoded in the step 1105 are displayed based on thedisplay time obtained in the step 1104.

The procedure taken at the time of determining samples to be decodedwill be described below with reference to the respective Error samplebox, Error free end sample box and Error free sequence box in the step1102.

First, at the time of referring to an Error sample box, it is determinedthat samples whose data are lost are not decoded. Note that, in the casewhere data loss information can be obtained on a basis of NAL unit of aslice, it is possible to determine whether or not samples should bedecoded on the basis of slice NAL unit of a sample. Also, in the casewhere whether or not the data of the slice NAL unit referred to at thetime of decoding the slice NAL unit of a sample is shown, it is possibleto decode the slice NAL unit data only in the case where all the data ofthe slice NAL units as reference destinations are completely available.For example, in the case where a coding type of slice NAL unitscomposing each sample periodically changes to another coding type, it ispossible to identify the data of samples referred to by the slice NALunit on condition that the periodical structure has already known. Hereis an example: a coding type periodically changes to another coding typeevery 15 samples, and the coded type of the slice NAL unit in each ofsamples in a period is IBBPBBPBBPBBPBB in a decoding order. Here, “I”shows an intra slice or a sample composed of IDR slices, “B” shows asample composed of bi-directionally-predictive slices and “P” shows asample composed of one-directionally-predictive slices. At this time, oncondition that a B sample refers to two samples except the B sampleplaced immediately before in decoding order and P sample refers to asample except the B sample placed immediately before in decoding order,it is possible to identify the sample referred to by the slice data of aB sample and a P sample.

Next, it is determined that samples outside an error-free area are notdecoded when samples to be decoded are determined with reference to anError free end sample box.

Lastly, when samples to be decoded are determined with reference to anError free sequence box, in the case where samples whose data is lostare included in the two continuous Sync samples, only the first Syncsample is decoded.

Note that, in the case where two or more boxes of Error sample box,Error free end sample box and Error free sequence box are concurrentlyused, it is possible to change the determination method of samples to bedecoded depending on the box that is being used. For example, in thecase where an Error sample box and an Error free end sample box are usedconcurrently, samples to be decoded correctly among samples outside theerror-free area are shown by the information of an Error sample box. Atthis time, it is possible to decode samples outside the error-free area.The same is true of the case where an Error sample box and an Error freesequence box are concurrently used.

Also, in this embodiment, the loss information notification unit 406 cinquires the user of whether or not the MP4 file should be reproduced,but another or no inquiry may be made. For example, the loss informationnotification unit 406 c may simply notify the user of the informationsuch as loss rate of sample data instead of inquiring the user ofwhether the MP4 file is reproduced or not. Also, the loss informationnotification unit 406 c may judge the information such as loss rate ofsample data using a predetermined threshold and, in the case where thecondition is satisfied, output reproduction indication signal to thesample obtainment unit 404. Further, the loss information notificationunit 406 c may judge the information such as loss rate of sample datausing a predetermined threshold and, in the case where the condition isnot satisfied, inquire the user of whether the MP4 file is reproduced ornot.

Note that the multiplex scheme conversion apparatus can also reproduce aconventional MP4 file.

Fifth Embodiment

The demultiplexer concerning the fifth embodiment of the presentinvention will be described with reference to FIG. 28. The demultiplexerinputs the MP4 file made by the multiplex scheme conversion apparatus inthe first to third embodiments, multiplexes, decodes and displays the AUdata. The demultiplexer determines decoding or displaying operations byreferring to the error information shown in the NAL unit for errornotification inserted into the sample data. Note that a Box indicatingerror information does not need to be added to the MP4 file to beinputted to the demultiplexer. The operation performed at the time ofreproducing a ‘trak’ of the MPEG-4 AVC will be described below, but a‘trak’ to be reproduced is not limited to the MPEG-4 AVC. It may be thevideo coded data such as H. 263 or the MPEG-4 visual, audio coded datasuch as the MPEG-4 AAC or AMR or coded data that reproduces ‘trak’scomposed of video and audio data concurrently.

The structure of this demultiplexer is the same as the structure of thedemultiplexer of the fourth embodiment shown in FIG. 25 except that itdoes not have the error information analysis unit 405 and it is notdescribed here. FIG. 28 is a flow chart showing the operation of thisdemultiplexer. First, the Sync sample from which decoding is started isdetermined based on the reproduction starting time inputted in the step1101. The processing of Loop A is performed after the processing of step1101 is completed. The Loop A indicates repeating the processing of step1102 to 1105, and it is repeated until the processing of the last sampleto be decoded in reproduction is completed. In the step 1102, the dataof samples to be decoded is obtained. The following processing of step1103 to step 1105 is performed by the decode and display unit 406. Inthe step 1103, a NAL unit for error notification and a sample isidentified in a sample data obtained in the step 1102 and in the casewhere it is shown that the sample is not correctly decoded, the sampleis not decoded. Consequently, the data of the slice NAL unit to bedecoded, such slice NAL data being determined in the step 1103, isdecoded in the step 1104, and in the step 1105, the decoded sample datais displayed based on the display time.

Sixth Embodiment

Here, the system where the multiplex scheme conversion apparatus and ademultiplexer shown in the above-mentioned first to fifth embodimentswill be described.

FIG. 29 is a block diagram showing the whole structure of the system forrealizing a content distribution service through broadcasting andcommunication. First, the case of receiving broadcasting data will bedescribed. A mobile phone ex101 or a disc recorder ex104 such as a DVDrecorder receives a TS packet sequence where coded digital media data ismultiplexed. The mobile phone ex105 converts the received TS packetsequence into an MP4 by using its internal multiplex scheme conversionapparatus in the present invention and then records it in an SD cardex106. The recorded MP4 file can be viewed or listened to by a mobilephone ex105 having a demultiplexer in the present invention, a discrecorder ex104 or a personal computer that is not shown. Also, it ispossible to view and listen to the MP4 file by attaching it to electricmail and, via the mobile phone ex105 to the wireless base station ex107,sending it to another mobile phone ex108 having a demultiplexer in thisinvention. Furthermore, it may be downloaded from the mobile phone ex105to the mobile phone ex108 or distributed in pseudo-streaming by using aprotocol such as the Hyper text transport protocol (HTTP) and theTransmission control protocol (TCP) instead of attaching it to electricmail.

Also, the disc recorder ex104 can make its internal multiplex schemeconversion apparatus in the present invention to convert the received TSpacket sequence to the MP4 and record it in an SD card, an optical discsuch as a DVD and a hard disc. Also, it may distribute, in download orin pseudo-streaming, the recorded MP4 file to a mobile phone or apersonal computer that is not shown in any figure. Note that it is alsopossible to record a TS packet sequence in an SD card and convert it toan MP4 file in distribution.

Further, it is possible to use the MP4 file, like in the case where theabove-mentioned broadcasting data is received, also in the case wherethe TS packet sequence distributed from the content server ex102 via theInternet are received by the mobile phone ex105, or the disc recorderex104.

Seventh Embodiment

It becomes possible to easily perform, in an independent computersystem, the processing shown in the embodiments that are respectivelymentioned in the above description by recording, in a storage mediumsuch as a flexible disc, (i) the multiplex scheme conversion methodperformed in the mutiplex format conversion apparatus shown in theembodiments respectively described above and (ii) a program forrealizing the demultiplex scheme performed in the demultiplexer.

FIG. 30A to 30C are an illustration of the case where the multiplexscheme conversion method in the multiplex scheme conversion apparatusand the demultiplex scheme in the demultiplexer in the respectiveembodiments are performed in a computer system using a program recordedin a recording medium such as a flexible disc.

FIG. 30A shows an example of the physical format of a flexible disc as arecording medium body. FIG. 30B shows a flexible disc and the front viewand the cross-sectional view of the appearance of the flexible disc. Aflexible disc (FD) is contained in a case F, a plurality of traks (Tr)are formed concentrically on the surface of the disc from the peripheryinto the inner radius of the disc, and each trak is divided into 16sectors (Se) in the angular direction. Therefore, in the case of theflexible disc storing the above-mentioned program, the program isrecorded in an area allocated for it on the flexible disc (FD).

Also, FIG. 30C shows the structure for recording and reading out theprogram on the flexible disc (FD). When the program for realizing themethod for converting multiplex scheme in the multiplex schemeconversion apparatus and the demultiplex scheme performed in thedemultiplex apparatus is recorded on the flexible disc (FD), the programis written on a flexible disc by the disc drive of the computer system(Cs). Also, when the multiplex scheme conversion method in the multiplexscheme conversion apparatus and the demultiplex method in thedemultiplexer are configured in the computer system by the program onthe flexible disc, the program is read out from the flexible discthrough a flexible disc drive and transferred to the computer system.

Note that a flexible disc is described as a recording medium above, butthe recording medium is not limited to a flexible disc, and an opticaldisc may be described as a recording medium in a similar way. Morespecifically, the recording medium is not limited to this, any recordingmedium such as IC cards and Rom cassettes are also available.

Although only some exemplary embodiments of this invention have beendescribed in detail above, those skilled in the art will readilyappreciate that many modifications are possible in the exemplaryembodiments without materially departing from the novel teachings andadvantages of this invention. Accordingly, all such modifications areintended to be included within the scope of this invention.

INDUSTRIAL APPLICABILITY

The multiplex scheme conversion apparatus in the present invention isapplicable for apparatuses that convert, to an MP4, the coded data suchas the MPEG-4 AVC that is sent by the TS, and record or send it. Also,this multiplex scheme conversion apparatus is particularly effective formobile phones that receive digital broadcasting.

1. A multiplex scheme conversion apparatus that (i) packet-multiplexesthe coded media data using a second scheme which is different from afirst scheme by which the coded media data has been packet-multiplexedand (ii) outputs the coded media data that is packet-multiplexed usingthe second scheme, said multiplex scheme conversion apparatuscomprising: a packet loss judgment unit operable to judge whether or nota packet loss has occurred in the coded media data that ispacket-multiplexed using the first scheme; a loss information makingunit operable to make loss information concerning a loss of data, basedon a position at which the data is lost in the coded media data, in thecase where said packet loss judgment unit is operable to judge that apacket loss has occurred; and a multiplex unit operable to store theloss information made by said loss information making unit in a packetthat is multiplexed using the second scheme and to packet-multiplex thecoded media data using the second scheme.
 2. The multiplex schemeconversion apparatus according to claim 1, further comprising a packetseparation unit operable to separate a header from a payload of a packetof the coded media data that is packet-multiplexed using the firstscheme.
 3. The multiplex scheme conversion apparatus according to claim2, wherein said packet loss judgment unit is operable to judge whetheror not a packet loss has occurred based on the header separated by saidpacket separation unit.
 4. The multiplex scheme conversion apparatusaccording to claim 2, further comprising a frame separation unitoperable to detect, in the payload separated by said packet separationunit, a boundary of frames in the coded media data and separate theframes, wherein said loss information making unit is operable to makeloss information concerning a loss of data on a frame-by-frame basisbased on a position at which the data is lost in the coded media data inthe case where said packet loss judgment unit is operable to judge thata packet loss has occurred, and said multiplex unit is operable to storeloss information made, on a frame-by-frame basis, by said lossinformation making unit, in the packet that is multiplexed using thesecond scheme.
 5. The multiplex scheme conversion apparatus according toclaim 4, wherein said loss information making unit is operable to make,as loss information, information indicating the area including frames(i) at which data necessary for decoding are not lost and (ii) that areplaced continuously in a decoding order.
 6. The multiplex schemeconversion apparatus according to claim 5, wherein said loss informationmaking unit is operable to make information concerning a leading frameof the area, as the information indicating the area.
 7. The multiplexscheme conversion apparatus according to claim 5, wherein said lossinformation making unit is operable to make information concerning alast frame of the area, as the information indicating the area.
 8. Themultiplex scheme conversion apparatus according to claim 4, furthercomprising a table making unit operable to make a table for registeringa randomly-accessible frame, wherein said multiplex unit is operable tojudge whether or not data necessary for decoding a frame is lost in theframe, based on a position at which data is lost in the coded mediadata, at the time of registering, in the table, a randomly-accessibleframe in the coded media data, and, in the case where data necessary fordecoding a frame is lost in the frame, said multiplex unit is operablenot to register, in the table, the frame as a randomly-accessible frame.9. The multiplex scheme conversion apparatus according to claim 4,further comprising a conversion unit operable to convert a storageformat of the coded media data.
 10. The multiplex scheme conversionapparatus according to claim 9, wherein the frame is composed of one ormore sub-frame units, and said conversion unit is operable to (i)determine sub-frame data to be stored in the packet that is multiplexedusing the second scheme and a size of the sub-frame data, based on theposition at which data is lost in the coded media data, and (ii) convertthe storage format by adding size information of the sub-frame unit tothe top of each sub-frame unit.
 11. The multiplex scheme conversionapparatus according to claim 9, wherein the frame is composed of one ormore sub-frame units, and said conversion unit is operable to insert thesub-frame unit indicating a loss of data into the position at which datais lost in the coded media data and to packet-multiplex the coded mediadata using the second scheme.
 12. The multiplex scheme conversionapparatus according to claim 1, wherein said loss information makingunit is operable to make information indicating whether or not datanecessary for decoding a predetermined unit is lost, as the lossinformation.
 13. A demultiplexer that decodes packet-multiplexed data ofcoded media data, comprising: a loss information extraction unitoperable to extract, from the multiplexed data, loss informationconcerning a loss of data; a loss information analysis unit operable todetermine data to be decoded in the coded media data, based on the lossinformation extracted by said loss information extraction unit; and adecode unit operable to decode the data to be decoded determined by saidloss information analysis unit.
 14. The demultiplexer according to claim13, further comprising a loss information notification unit operable tomake a notification concerning reproduction quality to a user based onthe loss information extracted by said loss information extraction unit.15. The demultiplexer according to claim 14, wherein said lossinformation notification unit is operable to receive an indication ofthe user based on the notification.
 16. A multiplex scheme conversionmethod for (i) packet-multiplexes the coded media data using a secondscheme which is different from a first scheme by which the coded mediadata has been packet-multiplexed and (ii) outputs the converted codedmedia data that is packet-multiplexed using the second scheme, saidmultiplex scheme conversion method comprising: judging whether or not apacket loss has occurred in the coded media data that ispacket-multiplexed using the first scheme; making loss informationconcerning a loss of data, based on a position at which the data is lostin the coded media data, in the case where it is judged that a packetloss has occurred; and storing the loss information made, in saidmaking, in a packet that is multiplexed using the second scheme andpacket-multiplexing the coded media data using the second scheme.
 17. Aprogram for causing a computer to (i) packet-multiplex the coded mediadata using a second scheme which is different from a first scheme bywhich the coded media data has been packet-multiplexed and (ii) outputthe coded media data that is packet-multiplexed using the second scheme,said program comprising: judging whether or not a packet loss hasoccurred in the coded media data that is packet-multiplexed using thefirst scheme; making loss information concerning a loss of data, basedon a position at which the data is lost in the coded media data, in thecase where it is judged that a packet loss has occurred; and storing theloss information made, in said making, in a packet that is multiplexedusing the second scheme and packet-multiplexing the coded media datausing the second scheme.